Skip to content

Add guidance for capturing ETW traces in Kubernetes pods#2344

Merged
brianrob merged 4 commits intomainfrom
copilot/add-guidance-for-traces-in-k8s
Dec 3, 2025
Merged

Add guidance for capturing ETW traces in Kubernetes pods#2344
brianrob merged 4 commits intomainfrom
copilot/add-guidance-for-traces-in-k8s

Conversation

Copy link
Contributor

Copilot AI commented Dec 1, 2025

  • Explore the repository structure and existing documentation
  • Understand the existing Windows containers section in UsersGuide.htm
  • Research EnableEventsInContainers and ImageIDsOnly options
  • Add new section "Capturing ETW Traces with Process-Isolation Windows Containers (Kubernetes)" to UsersGuide.htm
  • Address PR review feedback:
    • Clarify process-isolation is the default mode
    • Add note that Hyper-V mode doesn't require these instructions
    • Fix kernel session limitation explanation
    • Remove "(Required)" from Step 1 heading
    • Add that host user-mode events still work without /EnableEventsInContainers
    • Add note about building PerfViewCollect from source
    • Fix PDB signature terminology
    • Update error example to show "module!?" format
    • Clarify jitted code still works
    • Explain merge component container access limitation
  • Run code review
Original prompt

This section details on the original issue you should resolve

<issue_title>Guidance on How to Capture Traces in Kubernetes Pods</issue_title>
<issue_description>> Hi @brianrob. I'm currently running into a similar issue trying to run PerfView in a k8s pod. For those less knowledgeable with this tool is it possible to provide more context of what to do in this scenario?

Is the expectation on the Host node a user manually starts the PerfView process to collect a trace using the /EnbleEventsInContainer parameter. After they are done collecting the trace you then copy the zip into the container and perform a PerfView merge -ImageIDsOnly NameOfTrace.etl.zip? What does -ImageIDsOnly do exactly? I run that command and can't really tell if it worked.

If you don't run -ImageIDsOnly is it still possible to use that trace to profile what occurred in the k8s pod?

If this should be a new issue or discussion please let me know. Thanks!

Originally posted by @Cjewett in #1790</issue_description>

<agent_instructions>Please add a section to the PerfView users guide on how to capture ETW traces when one or more processes are in process-isolation Windows containers. It would be especially helpful if you provide example command lines, and describe what each command does, as well as which ones are required and which ones are optional. If they're optional, describe what happens if you don't run them.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@brianrob There are some strong limitations in how we can best collect ETW traces on Windows when containers are involved. The happy path is the following:
  1. Capture an ETW trace on the host node. Specify /EnableEventsInContainers when collecting to ensure that user-mode events flow from processes inside of containers back to the ETW session. If you don't specify this, then you'll get all kernel events but only user-mode events from processes running directly on the host and not inside of a container.
  2. At this point, as long as the container(s) containing the process(es) of interest are still running, you can open and analyze the trace on the host node. PerfView will be able to find any binaries that it needs both on the host and inside the containers.
  3. If you want to shutdown a container of interest, or copy the trace to another machine for analysis where the container is inaccessible, then you'll need to copy the trace into the container, and then inside the container, run PerfViewCollect merge -ImageIDsOnly <path/to/trace>. Once it completes, copy the resulting file (overwrites the original) out of the container. This command is responsible for opening all of the accessible binaries that have symbols in the trace, and injects PDB information into the trace, so that we can download the PDBs from the symbol server later on during analysis. If you don't do this, then some symbols just won't be resolvable later. The data is still valid, but might be less useful.</comment_new>
    <comment_new>@brianrob
    @Cjewett, let me know if you have any questions on this. I'm also going to use this issue to get copilot to update the PerfView users guide with this information.</comment_new>

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: brianrob <6210322+brianrob@users.noreply.github.com>
Copilot AI changed the title [WIP] Provide guidance on capturing traces in Kubernetes pods Add guidance for capturing ETW traces in Kubernetes pods Dec 1, 2025
Copilot AI requested a review from brianrob December 1, 2025 20:38
Copy link
Member

@brianrob brianrob left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot, please address these comments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarify that process-isolation mode is the default and that if running in Hyper-V mode, these instructions are not required - just capture inside the container.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kernel sessions specifically cannot be started from inside of a container. PerfView almost always captures a kernel session, so we should assume so here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<h6>Step 1: Capture a Trace on the Host Node (Required)</h6>
<h6>Step 1: Capture a Trace on the Host Node</h6>

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will still get user-mode events from processes running directly on the host node (outside of containers).

Then, inside the container, run the merge command to inject the necessary image identification data:
</p>
<ul>
<li>PerfViewCollect merge /ImageIDsOnly MyContainerTrace.etl.zip</li>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: PerfViewCollect needs to be built from source at https://github.com/microsoft/perfview. It is not currently shipped as a binary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It actually looks up the PDB signature.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stacks will just show the module name with a question mark. For example: MyAssembly!? instead of MyAssembly!MyClass.MyMethod.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jitted code will still work, but nothing else will that comes from a binary inside of the container.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The merger is a shared component and doesn't have access to look inside of containers, thus you must run it inside of the container(s).

Co-authored-by: brianrob <6210322+brianrob@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Dec 1, 2025

@copilot, please address these comments.

Addressed all review comments in commit 15fe2b2.

Copilot AI requested a review from brianrob December 1, 2025 20:53
@brianrob brianrob marked this pull request as ready for review December 3, 2025 00:56
@brianrob
Copy link
Member

brianrob commented Dec 3, 2025

Copilot Post-Mortem:

  • As expected, it was easy to have copilot take my bullet points and write a more complete document.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Guidance on How to Capture Traces in Kubernetes Pods

3 participants

Comments